ENGINEERING LEARNING: MULTIPLE INFLUENCES 
ON THE DEVELOPMENT OF ANALYTICAL 
AND GROUP SKILLS 



Linda C. Strauss 
Senior Project Associate 
Center for the Study of Higher Education 
Pennsylvania State University 
University Park, PA 16802 
Voice: (814) 863-2655 
Fax: (814) 865-3638 
E-mail: lus2@psu.edu 

Patrick T. Terenzini 

Distinguished Professor and Senior Scientist 
Center for the Study of Higher Education 
Pennsylvania State University 



Paper presented at the meeting of the Association for Institutional Research, June 2005, 
San Diego, CA. 

This study was supported by a grant from the Accrediting Board of Engineering and 
Engineering Technology (ABET). The opinions expressed here do not necessarily reflect 
the opinions or policies of ABET, and no official endorsement should be inferred. 

The authors wish to acknowledge the contributions of the members of the research team 
who participated in the development of the design, instruments, and databases for this 
project: Dr. Lisa R. Lattuca, project director and co-PI; Dr. J. Fredericks Volkwein, co- 
PI; Jazvan Suhkbaatar, graduate research assistant; Vicki L. Baker, graduate research 
assistant, and Suzanne Bienart, project assistant. 



Multiple Influences - 2 



ENGINEERING LEARNING: MULTIPLE INFLUENCES 
ON THE DEVELOPMENT OF ANALYTICAL 
AND GROUP SKILLS 

Abstract 

This paper reports the results of a study exploring the unique and joint 
contributions of engineering students’ classroom and out-of-class experiences on the 
development of two sets of skills central to students’ successful performance as 
engineers: problem-solving and analytical skills, and group skills. Although the study 
focuses on engineering, the criterion measures and conceptual underpinnings are relevant 
to studies of teaching and learning in other fields. Multiple regression analyses using 
data from more than 4,500 graduating engineering students on 39 campuses nationwide 
indicate that, after controlling an array of students’ precollege characteristics, both 
students’ classroom and out-of-class experiences make statistically significant and unique 
contributions to student learning in each of these skill areas. 




Multiple Influences - 3 



ENGINEERING LEARNING: MULTIPLE INFLUENCES 
ON THE DEVELOPMENT OF ANALYTICAL 
AND GROUP SKILLS 



In the early 1990s, the Accreditation Board of Engineering Education (ABET), 
responding to increasing mismatches between industry needs and the skills of 
engineering graduates, as well as to pressures within the engineering education 
community, adopted a set of criteria for evaluating engineering programs nationwide that 
was dramatically different from the previous criteria (Engineering Accreditation 
Commission, 1998). The new model, referred to as “EC2000” (shorthand for the title of 
the document specifying the new criteria), shifted the emphasis of the re-accreditation 
review away from the possession of specific resources and facilities to specific learning 
outcomes. The new criteria required that engineering programs show evidence that their 
graduates were competent in 11 specific skill areas. For example, ABET now requires 
institutions to demonstrate (among other things) that their graduates can “ design a 
system, component, or process to meet desired needs” and “function on multidisciplinary 
teams” (Engineering Accreditation Commission, 1998). 

While consensus exists about what competencies undergraduate engineering 
students should develop, less clarity and consensus surround the question of how to help 
students effectively develop those competencies. Instructional staff members in all kinds 
of institutions and fields are under pressure to enhance the effectiveness of their teaching. 
The relationship between students’ classroom experiences and learning in undergraduate 
engineering (and quite possibly in other fields as well) less-well understood than one 
might expect. Dutson, Todd, Magleby, and Sorensen (1997) reviewed more than 100 
papers and articles relating to engineering design courses. The majority of those 
publications dealt with issues such as course development, structure, characteristics, 
faculty roles, design project characteristics, industrial involvement, and the composition 
and activities of student design teams. With respect to how much students learned, 
Dutson et al. concluded “The literature is filled with positive comments from students, 
instructors, and industrial sponsors,” but “The nature of capstone design courses. . . 
often leads to a purely subjective evaluation with little or no ‘hard evidence’ of actual 
benefits” (p. 24). The irony of this is not merely the scarcity of studies of what kinds of 
experiences promote engineering learning, but also that the research literature approaches 
learning as if such learning could be promoted only, or primarily, in the classroom. This 
latter belief, unfortunately, is common across disciplines and professions. The view that 
student academic and cognitive learning occurs primarily (if not exclusively) in the 
classroom is broadly held, but it has little foundation in the research on how students 
develop academically and cognitively (Pascarella & Terenzini, 2001, 2005). 

This paper explores an array of student experiences, some of which are commonly 
overlooked, that are designed to promote the learning outcomes specified in EC2000. 
Specifically the paper describes the findings of a study examining the unique and joint 
contributions of classroom and out-of-class learning environments on students’ 
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development of their abilities to analyze unstructured engineering problems (a special 
case of what is commonly called post-formal reasoning) and to work successfully in 
groups. 



A review of the engineering education literature reveals few studies that examine 
the contribution of students’ out-of-class experiences, as well as traditional classroom 
curricula and courses, to student learning. In their review of the literature on out-of-class 
influences on cognitive development, Terenzini, Pascarella, and Blimling (1995) 
examined seven types of out-of-class activities to estimate the impact each has on 
learning and higher-order skill development. Their review suggests that living in a 
residence hall (Blimling, 1989), not belonging to a fraternity of sorority (Astin, 1993; 
Blimling, 1989; Pascarella, Edison, Whitt, Nora, Hagedom, & Terenzini, 1996), working 
part-time on campus (Astin, 1993), involvement in other clubs and student organizations 
(Terenzini, Springer, Pascarella, and Nora, 1995a), study abroad programs (Astin, 1993), 
faculty interaction outside of the classroom (Astin, 1993; Terenzini, Springer, Pascarella, 
& Nora, 1995a, 1995b), tutoring (Astin, 1993), and socializing with peers from different 
racial/ethnic groups (Astin, 1993; Kuh, 1995) all contribute positively to learning and 
cognitive development. The review also identified some positive benefits to participation 
in intercollegiate athletics (Astin, 1993; Pascarella & Smart, 1991), although other studies 
did not support these finding (Pascarella, Bohr, Nora, & Terenzini, 1995; Pascarella, 
Truckenmiller, Nora, Terenzini, Edison, & Hagedorn, 1999). Terenzini et al. (1995) 
concluded that cognitive development is more likely a function of the cumulative effects 
of multiple experiences than the product of any particular experience. 

Kuh, Hu, and Vesper (2000) also concluded that involvement in any one out-of- 
class activity does not maximize overall learning gains. Building on the typology of 
college students developed by Clark and Trow (1966) in the 1960s, Kuh et al. formed ten 
student types by examining the kinds of activities in which students engaged. They 
found that student types characterized by heavy involvement in only one or two activities 
exhibited average or, more frequently, below-average gain scores, whereas student types 
involved in an array of activities tended to benefit most from their collegiate experience. 

While the literature reviewed above supports the proposition that out-of-class 
activities positively influence academic outcomes, the kinds of activities run the gamut 
from experiences directly related to classroom material to those focusing on personal and 
social interactions. Kuh (1995) examined specific out-of-class activities and how they 
related to specific outcomes. His qualitative study included several out-of-class examples 
specifically tied to academic areas. Independent research opportunities provided 
opportunities for students to enhance the knowledge they had learned in the classroom. 
For example, serving as a student government leader enabled one student to learn how to 
plan budgets and manage resources. A weekend job provided another student with 
opportunities to apply knowledge gained about production techniques. Kuh concluded 
that out-of-class experiences comprise the “real world” laboratory, and that institutions 
need to view them as part of, rather than separate from or competing with, traditional 
curricula and classroom. 
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Terenzini, Springer, Pascarella, and Nora (1995a, 1995b) undertook to partition 
the unique and joint contributions of both students’ out-of-class activities and in-class 
activities on two measures of higher-order cognitive skills, critical thinking and 
intellectual orientation. After taking into account students’ precollege characteristics, 
these researchers found that in- and out-of-class experiences were both significant and 
independent contributors to students’ critical thinking and intellectual development. For 
both outcomes, students’ out-of-class experience contributed less than their in-class 
experiences, but in both instances these experiences were still statistically significant 
influences. These analyses also revealed the joint contribution of students’ in- and out- 
of-class experiences was statistically significant, even after controlling all other variables 
in the model. 

Although the innovation of isolating the unique and shared effects of out-of-class 
and in-class variables on student learning outcomes in the Terenzini et al. studies are 
noteworthy, these studies focus on general forms of learning, not on program-related 
learning goals. In addition, these studies are based on a single-institution sample with 
students from a variety of majors, leaving differences in academic programs 
uncontrolled. This study builds on the work of Terenzini et al. by using multi- 
institutional data and limiting the scope of “learning” to that of a single academic major. 
Engineering was selected because the skill sets under examination are specific and widely 
endorsed by both scholars and practitioners (industry) within the field. In their emphasis 
on unstructured problem-solving and group skills, they are also generalizable to some 
degree to fields other than engineering. 

METHODS 



Conceptual Framework 

Figure 1 portrays the hypothesized relationships among the new EC2000 
accreditation standards, changes in engineering programs, and student learning. The 
conceptual framework for the overall EC2000 project suggests that changes in student 
learning will be a product of EC2000-induced modifications in the curricula and 
instructional practices engineering programs offer, in the engineering faculty culture, and 
in the college of engineering and program administrative policies and practices. 
Assuming the implementation of the EC2000 criteria has, in fact, had an influence on 
engineering education and student learning, these curricular, instructional, cultural, and 
organizational shifts are all presumed to be consistent with the goals of the EC2000 
accreditation standards. The linkages are, in this sense, indirect: preparation for an 
EC2000 accreditation results in curricular and other changes that, in turn, affect student 
learning. 

— Place Figure 1 About Here - 



This study focuses on one segment of the overall conceptual framework, namely, 
that relating to the effects of varying instructional practices and students’ out-of-class 
experiences on students’ engineering design and analytical skills and their abilities to 
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work in teams. This set of propositions suggests that students’ development of these 
skills is a function of both what occurs in the classroom and of the kinds of out-of-class, 
engineering -related activities students have over the course of their programs. 

Design, Population, and Sample 

Because of limitations on time and resources, the research group adopted a cross- 
sectional, post hoc survey design. The study is part of a national study designed to 
explore the impact of the implementation of the EC2000 criteria on the preparation of 
engineering graduates to enter their profession. The overall project contrasts graduates of 
undergraduate engineering programs both pre- and post-EC2000 implementation, 
although the present research uses data only from the post-EC2000 sample. As will be 
seen below, this study concentrates on one portion of the educational process presumably 
set in motion by the introduction of the EC2000 outcomes criteria. 

The study’s population includes all ABET- accredited engineering programs. The 
target population was defined to include those programs accredited by ABET since 1990 
in selected fields. The project targets graduates of programs in seven engineering 
disciplines: aerospace, chemical, civil, computer, electrical, industrial, and mechanical 
engineering. This disciplinary array includes both those disciplines that produce the vast 
majority of engineering graduates in any given year (chemical, civil, electrical, and 
mechanical), as well as disciplines with strong ties to industry sectors (aerospace, 
computer, and industrial). 

Of the population of 1,241 ABET-accredited engineering programs in the targeted 
disciplines, 1,024 met the accredited- since- 1990 specification. The project team selected 
programs for participation in the study based on a two-stage, 7x3x2, disproportionate, 
stratified random sample. In the first stage, institutions in the target population were 
stratified on three criteria: 1) they contained at least two of the targeted seven disciplines, 
2) three “EC2000-adoption” groups (i.e., were reviewed under EC2000 before being 
required to do so, reviewed when mandatory, and elected to defer EC2000 review when 
that option was available), and 3) whether the programs and institutions had participated 
in a National Science Foundation Engineering Education Coalition during the 1990s 
(these coalitions were among the leaders in moving toward meeting the EC2000 criteria). 
The sample is “disproportionate” in its over-sampling of smaller disciplines (aerospace 
and industrial) to ensure an adequate number of responses for analysis. To ensure a 
representative sample of institutions, four EC2000 pilot institutions (first reviewed in 
1996 and 1997) were also included, as were several Historically Black Colleges and 
Universities (HBCUs) and Hispanic Serving Institutions (HSIs). The final sample 
included 203 programs at 39 institutions. 

Data Collection Procedures 

The main project’s design entailed data collection from several sources, including 
graduating seniors, alumni, faculty members, program chairs, deans, and employers. 
Recent undergraduate students provided the data for the current study. In spring, 2004, 
the population of 12,621 seniors nearing graduation in any of the seven targeted 
engineering fields on 39 nationally representative campuses selected according to the 
sampling design described above were sent the final survey instrument (available at 
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[URL deleted due to institutional reference]) (one of the original 40 institutions did not 
provide student responses). The Graduating Seniors Survey solicited information on 
basic demographic information, level of participation in out-of-class activities related to 
engineering education, student learning outcomes associated with each of the 1 1 EC2000 
outcomes criteria, classroom practices, and plans for the future. 

Hard copies of the survey instrument (and the URL for an electronic version) 
were sent to students at 20 of the institutions, and electronic versions only were sent to 
the students at the other 20 institutions. The dean of the college of engineering on each 
campus signed the cover letter in both administrations. Follow-up waves included a 
postcard sent two weeks after the initial mailing and a complete follow-up (similar to the 
initial mailing) sent two weeks after the postcard. 

A total of 4,558 (36.1%) of the seniors completed and returned the survey. Any 
case with more than 20 percent of the possible responses missing was deleted from the 
database, yielding 4,330 usable cases (a usable response rate of 34%). Missing data in 
the remaining cases was imputed using expected maximization method (Allison, 2001). 

Variables 

The criterion measures in this study are two of nine factorially derived scales 
produced in a lengthy, detailed process to operationalize the specifications of the EC2000 
learning outcomes (Author names withheld, 2005). These nine scales were formed using 
a series of principal components analyses with varimax rotations. Table 1 summarizes 
the results of these analyses and subsequent internal consistency (Cronbach’s alpha) 
reliability analyses. The final, nine-factor structure retains 72.2 percent of the overall 
item variance among the original 36 survey items. All 36 items loaded above .40 on a 
single factor, and none loaded above .40 on any two factors. The factor scale scores were 
calculated by summing students’ responses to each item loading above .40 on a 
component and dividing by the number of items in the scale (Armor, 1974). The scale 
alphas were above .83 on all but two scales. On those scales, the alphas were .74 and .78. 
As noted, only two of the nine scales are of interest to this study: the Design and 
Analytical Skills Scale (six items; alpha = .92) and the Group Skills Scale (three items; 
alpha = .86). The Design and Analytical Skills scale taps, essentially, students’ abilities 
to solve unstructured engineering problems (i.e., those for which there is no single or 
“correct” solution). The Group Skills scale reflects students’ abilities to work 
successfully with others to accomplish a team goal. 

— Insert Table 1 About Here — 

Because the study was concerned with identifying student activities and 
experiences over which faculty and administrators have some programmatic or policy 
control, potentially confounding precollege student characteristics were controlled. 
Individual differences used as covariates included students’ age, gender, high school 
preparation in basic math and science. These variables are listed in Table 2. 
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The independent variables in this study fall into three sets: 1) the control 
variables: students’ precollege characteristics (e.g., preparation for college, age, and 
family income), and two sets of independent variables of primary interest: 2) the 
instructional practices students reported faculty used in the classroom, and 3) students’ 
reports of their out-of-class activities in several areas relevant to engineering education. 
These variables are listed in Table 2. 

— Insert Table 2 About Here — 

The instructional practices set of variables included three, factorially derived 
(principal components with varimax rotation) scales: Clarity and Organization, 
Collaborative Learning, and Instructor Interaction and Feedback. The component items 
for these scales are given in Table 2. Each scale was formed by summing its component 
items (loadings above .40) and dividing by the number of items in the scale. These scales 
consist of three, seven, and five items respectively, with alphas of .82, .90, and .87. 
Students’ reports of their involvement in seven out-of-class experiences were also 
employed (e.g., cooperative education experiences, study abroad, and employment during 
college). 

Analytical Procedures 

Data were analyzed using hierarchical (or blocked) ordinary least-squares (OLS), 
multiple regression. Two regression models were run. In both, the first block entered in 
the model included students’ precollege characteristics. In the first regression, the 
instructional practices variables were entered ahead of students’ out-of-class experiences. 
In the second regression, the order was reversed, with the instructional practices block of 
variables being entered after students’ out-of-class experiences. These two models were 
used for both criterion measures. 

The unique variance attributable to each set, as well as the variance jointly 
explained by the two sets of primary independent variables, was estimated by the 
following process: 

A = B+C+D 

B = A (with C and D absent from model) 

C = (A - B - D) 

D = (A - B - C), and 

E = A - (B+C+D), where 

A = Total variance explained 

B = R 2 C hange due to precollege characteristics (the covariates) 

C = R 2 C hange due to entry of Instructional Practices set when model already 

includes B and D (i.e., R 2 c han g e when set was entered on the last step) 

D = R 2 c hange due to Out-of-Class Experiences when model already includes B and 
C (i.e., R 2 C hange when set was entered on the last step) 
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2 

E = R“ jointly explained by Instructional Practices and Out-of-Class Experiences 
(i.e., variance unattributable uniquely to B or to either C or D). 

RESULTS 

Table 3 summarizes the results of the two regression analyses. With the Design 
and Analytical Skills scale as the dependent variable, the overall model produced an 
adjusted R 2 of .172 (pc.OOl). Although the R 2 may seem small, it is worth noting that 
that estimate is based on student reports of the extent to which they encountered various 
instructional approaches throughout their four-year undergraduate engineering career. 

The reports are not specific to individual courses, where stronger relationships between 
classroom experiences and outcomes might be expected. 

— Insert Table 3 About Here — 

The findings also indicate that, after controlling for precollege differences among 
students, both their classroom and out-of-class experiences made statistically significant 
and unique contributions to student end-of-program reports of their engineering design 
and analytical skills. With controls in place for students’ precollege characteristics and 
their out-of-class experiences, students’ classroom experiences explained an additional 
7.2 percent (R 2 c han g e = -072, pc.OOl) of the variance in their reported end-of-program 
design and analytical skills. After taking into account students’ precollege characteristics 
and their classroom experiences, students’ out-of-class experiences also made a 
significant and unique contribution of 3.1 percent (p < .001) to the explained variance for 
students’ end-of-program reports of their design and analytical skills. In the full model, 
several components within each set made statistically significant (p < .001) and 
independent contributions to students’ design and analytical skills. These predictors 
included all three of the Instructional Practices subscales (Clarity and Organization, 
Collaborative Learning, and Instructor Interaction and Feedback), as well as students’ 
out-of-class cooperative education experiences and participation in design competitions. 
Employment during college also had a significant and positive influence the development 
of students’ design and analytical skills (p < .01). 

It is particularly noteworthy that both in- and out-of-class experiences contributed 
independently and significantly to students’ skill development and that the two sets of 
experiences also made a small (1.7%), joint contribution to student learning. 

The model with Group Skills as the criterion variable yielded a smaller, but still 
statistically significant, overall adjusted R 2 (.142, pc.OOl). Once again, both classroom 
and out-of-class experiences produced statistically significant and unique contributions to 
student learning, as well as sharing a small portion of the explained variance. The unique 
contribution of students’ in-class experiences was, again, larger (.096, pc.OOl) than that 
of their out-of-class experiences (.012, pc.OOl) as measured by the changes in R 2 . In 
fact, the difference between the in-class and out-of-class experiences is greater in the 
group skills model (a difference of 84 percent) than in the design and analytical skills 
model (59 percent). Significant instructional predictors of students’ end-of-program 
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group skills levels are similar to those for their design and analytic skills, including their 
instructors’ clarity and organization of instruction and use of collaborative learning 
methods; the instructor feedback scale was non-significant in this model. Among 
students’ out-of-class experiences, outside employment was again a significant and 
positive influence, along with students’ involvement in cooperative education, 
engineering design competitions, and student professional organizations. 

As in the case of design and analytical skills, there is a small (1.3%) joint 
contribution by both in-and out-of-class experiences to group skills. This contribution is 
in addition to the unique contributions above. 

[.imitations 

This study, like other investigations, is limited in several ways. First, although 
the study was designed to capture significant aspects of both in- and out-of-class 
experiences students’ have over the course of the engineering programs, the conceptual 
framework and its operational form in this study may be underspecified, overlooking or 
inadequately representing important dimensions of the learning process. Since the focus 
is on engineering students and program-related experiences, the out-of-class domain was 
restricted to those dimensions considered to be immediately relevant to engineering 
education. A wide array of other out-of-class activities are known to influence student 
learning, including place of residence during college, co-curricular involvement, and 
attending cultural or artistic events (Pascarella & Terenzini, 2001, 2005). Moreover, 
other in-class experiences or conditions, such as library experiences, the gender of the 
instructor, and course content may influence student learning but were not included in 
this study. Indeed, the modest adjusted R 2 s may suggest that other influences on 
students’ design and analytical skills and their group competencies may also need to be 
considered. 

Second, the dependent constructs - design and analytical skills and group skills - 
are complex skill sets and not easily measured. Although the process used to create the 
scales used in this study (Author names withheld, 2005) was consistent with the canons 
of survey development, the resulting measures may reflect only some of the talents that 
are necessary to be proficient in these skill areas. Direct observations of these skills 
would be preferred but also difficult to obtain from over 4,000 students on 39 campuses 
across the country. 

Participants in the study were limited to graduating seniors in engineering. 
Although respondents were representative of the national population from which they 
come, generalization of the results of the study to students in other academic areas is 
probably limited by the specialized and technical nature of engineering as a field of 
study. However, previous studies (cited in the literature review) have demonstrated the 
importance of both in-class experiences and out-of-class activities to student learning. 
Consequently, a modicum of confidence can be placed in the validity of these results. 
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Finally, any study that relies on self-reports of student learning must acknowledge 
the potential weaknesses of such data. A growing body of empirical evidence, however, 
suggests that, under certain conditions (believed to exist in this study’s outcome 
measures), self-reported learning can be a reasonable proxy for other, more objective 
measures (Anaya, 1999; Bradburn & Sudman, 1988; Carini, O’Day, & Kuh, 2002; 
Converse & Presser, 1989; Hayek, Laing, Sawyer, & Noble, 1988; Pace, 1985; Pike, 
1995). Moreover, although self-reports have acknowledged limitations when compared 
with standardized tests, the latter also come with their own limitations, including 
availability, length, cost, administration requirements, and relevance to the outcomes of 
interest. 



DISCUSSION AND IMPLICATIONS 

This paper estimated the unique and joint contributions of engineering students’ 
classroom and out-of-class experiences on the development of two sets of skills central to 
students’ successful performance as engineers: problem-solving and analytical skills, and 
group skills. Although the study focuses on engineering, the criterion measures and 
conceptual underpinnings are also relevant to studies of teaching and learning in other 
fields. Multiple regression analyses using data from more than 4,330 graduating 
engineering students on 39 campuses nationwide indicate that, after controlling an array 
of students’ precollege characteristics, both students’ classroom and out-of-class 
experiences make statistically significant and unique contributions to student learning in 
each of these skill areas. Significant and positive individual influences on students’ 
design and analytical skills included both the instructor behaviors (e.g., clarity and 
organization, use of collaborative learning approaches, and interactions with and 
feedback to students) and students’ program-related out-of-class experiences (e.g., 
participation in cooperative education and in engineering design competitions, and 
employment). The pattern of individual in- and out-of-class contributors to skill 
development was similar for students’ group skills. 

These analyses provide moderate -to- strong evidence that while instructional 
activities clearly play an important role in student learning outcomes, students’ out-of- 
class experiences are also significant forces. Although this study concentrated on 
teaching and learning in engineering, it seems reasonable to suggest that the same 
dynamics might shape student learning in other academic fields. Although the criterion 
measures adopted for this study were specific to engineering, one might reasonably argue 
that they are a special case of broader intellectual, analytical, and interpersonal skills, 
including post-formal reasoning (solving unstructured problems for which there is no 
single, correct answer) and the ability to work in groups. Within the context of 
developing strategies to enhance student learning, these findings underscore the 
importance of adopting broader conceptions of the learning environment than are 
typically employed. Undergraduate learning (at least in engineering, but also probably in 
other fields) appears to have both classroom and out-of-class components. By not 
capitalizing on the out-of-class dimensions of teaching and learning, faculty members, 
administrators, and policy makers are missing a significant driver in student learning. 
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The kinds of in class and out-of-class activities for both learning outcomes appear 
to be similar. As might be expected, opportunities for students to work collaboratively in 
their courses enhance their reported ability to work in groups. That same activity, 
however, also enhances students’ design and analytical skills, lending additional evidence 
to the proposition that active learning assists in the development of a variety of desired 
student outcomes. Similarly, a well-organized and clear instructor enhances students’ 
reports of their skills in both outcome areas of interest in this study. The kinds of skills 
students obtain working on design projects and in cooperative education experiences 
appear to be consistent with group skills. And although these particular activities may be 
specific to engineering, they have in common with other fields the opportunity to take 
what is learned in the classroom and to apply it in practical settings. Such opportunities 
both extend and reinforce what may otherwise be an abstract, theoretical experience in a 
classroom. Pascarella and Terenzini (2005) note the particular potency of out-of-class 
experiences that complement and extend classroom learning. 

The underlying dynamic remains unclear, but when students work on projects and 
are employed in work in settings, it seems reasonable to suggest that they also probably 
refine and enhance their group skills. It is also logical to expect these experiences to 
contribute to students’ analytical and design skills. Both involve the application of 
academic content to engineering problems. More surprising, however, is the apparent 
lack of any significant relation between group skills and instructor interaction and 
feedback. It may be that, for the students in this study, the instructional domain is more 
one-on-one than group-oriented; perhaps the lecture/discussion approach to teaching still 
dominates engineering instruction. The significant contribution of participation in 
student professional organizations on group skills, however, is consistent with the type of 
leadership and teamwork present in most student organizations. 

The study’s findings that certain experiences both in- and out of class influence 
student learning in desired directions can guide faculty members, program heads, and 
deans in the review of existing programs and the development of new ones. The study’s 
findings call attention to specific dimensions of students’ program experiences that 
warrant particular attention in efforts to facilitate student learning. This study suggests 
that cooperative education opportunities, participation in design competitions, and 
employment in engineering-related positions all hold potential to extend and reinforce 
what students experience in their coursework. The findings also imply that such 
experiences warrant attention when college/department/program budgets are developed 
and resource allocation decisions are made. 

Finally, these findings have implications for institutional researchers and others 
responsible for (or potentially benefiting from) learning outcomes assessment and 
continuous improvement. If students’ out-of-class experiences are overlooked in 
outcomes assessment designs, the resulting findings will miss an important aspect of the 
student experience and provide an incomplete portrait of the nature and extent of student 
learning. Other professional accrediting agencies, moreover, are watching the ABET 
model to ascertain whether it “works” to produce organizational change and, ultimately, 
desired student learning outcomes. Part of ABET’s emphasis rests on the premise that 
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the engineers of tomorrow must be skilled in a broader array of competencies (group and 
teamwork skills are an example) than has been the case to-date. The results of this study 
suggest that students’ out-of-class experiences can play an important part in shaping these 
outcomes. Failure on the part of assessment professionals and accreditors to take account 
of those experiences and their impacts will provide an incomplete assessment of the 
learning taking place, as well as constrain efforts to maximize the learning opportunities 
programs can make available to their students. 
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Figure 1: Conceptual Framework for the Engineering Change Study 
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Table 1. Factor structure underlying items operationalizing EC2000 learning 
outcomes criteria. 



Highest Loading Items 


Number 
of Items 


Factor 

Loadings 


Scale 

Alpha 


Variance 

Explained 


1. Design and Analytical Skills 


6 




.92 


12.0% 


Design solutions to meet desired needs 




.78 






Apply systematic design procedures to open-ended problems 




.77 






Define key engineering problems 




.76 






Formulate a range of solutions to an engineering problem 




.75 






Understand essential aspects of the engineering design process 




.68 






Apply discipline-specific engineering knowledge 




.49 






2. Societal and Global Issues 


5 




.92 


11.0 


Understand contemporary issues (economic, environmental, political, etc.) 




.80 






Understand that engineering decisions and contemporary issues 




.79 






Understand the impact of engineering solutions in a societal context 




.77 






Use knowledge of contemporary issues to make engineering decisions 




.76 






Understand the impact of engineering solutions in a global context 




.76 






3. Codes and Ethics 


5 




.87 


8.4 


Understand the engineering code of ethics 




.79 






Consider ethical issues when working on engineering problems 




.78 






Work through ethical issues in engineering 




.78 






Understand technical codes and standards 




.56 






Conduct yourself professionally 




.48 






4. Experimental Skills 


4 




.89 


7.8 


Analyze evidence or data from an experiment 




.79 






Interpret results of an experiment 




.74 






Carry out an experiment 




.74 






Design an experiment 




.63 






5. Communication skills 


4 




.86 


7.8 


Convey ideas in writing 




.78 






Convey ideas verbally 




.76 






Convey ideas in formal presentations 




.73 






Convey ideas in graphs, figures, etc. 




.60 






6. Applying Engineering Skills 


4 




.94 


7.5 


Apply engineering tools in engineering practice 




.75 






Apply engineering skills in engineering practice 




.75 






Apply engineering techniques in engineering practice 




.73 






Integrate engineering techniques, skills, and tools to solve real-world problems 




.57 






7. Group Skills 


3 




.86 


6.9 


Work with others to accomplish team goals 




.85 






Work in teams of people with a variety of skills and backgrounds 




.83 






Work in teams where knowledge and ideas from multiple 
engineering disciplines must be applied 




.65 






8. Life-long learning 


3 




.78 


6.0 


To what extent are you motivated to acquire and apply new technologies 




.81 






To what extent are you willing to take advantage of new opportunities to learn 




.80 






To what extent are you able to learn and apply new technologies and tools 




.73 






9. Applying Basic Skills 


2 




.74 


4.6 


Apply knowledge of mathematics 




.84 






Apply knowledge of physical sciences 




.77 






TOTAL VARIANCE EXPLAINED 








72.2% 
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Table 2. Independent variables used in OLS regression analyses. 



Control Variables 

1. Students’ Precollege Characteristics 

■ Age 

■ Sex 

■ Family income 

■ Overall high school GPA 

■ Highest level of education attained by mother 

■ Highest level of education attained by father 

■ First-time student or transfer student 

■ Preparation for basic science and math courses when entering college 

Independent Variables 

2. Instructional practices utilized by faculty 

■ Clarity and Organization Scale (alpha = .82) 

■ Assignments and class activities were clearly explained 

■ Assignments, presentations, and learning activities were clearly related to 
one another 

■ Instructors made clear what was expected of students in the way of 

activities and effort 

■ Collaborative Learning Scale (alpha = .90) 

■ I worked cooperatively with other students on course assignments 

■ Students taught and learned from each other 

■ We worked in groups 

■ I discussed ideas with my classmates (individuals or groups) 

■ I got feedback on my work or ideas from my classmates 

■ I interacted with other students in the course outside of class 

■ We did things that required students to be active participants in the 

teaching and learning process 

■ Instructor Interaction and Feedback Scale (alpha = .87) 

■ Instructors gave me frequent feedback on my work 

■ Instructors gave me detailed feedback on my work 

■ Instructors guided students’ learning activities rather than lecturing or 

demonstrating the course material 

■ I interacted with instructors as part of the course 

■ I interacted with instructors outside of class (including office hours, 

advising, socializing, etc.) 
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Table 2. Continued. 



3. Student reports of their out-of-class activities relevant to engineering 

■ Full-time or part-time enrollment 

■ Student employment status 

■ Months as an intern or cooperative education student 

■ Months in a study abroad program 

■ Months spent traveling internationally (not study abroad) 

■ Months spent in student design projects beyond classroom requirements 

■ Activity in a student chapter of a professional organization 
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Table 3. Partitioning of variance results for Design and Analytical Skills and Group 
Skills scales. 





Design and 
Analytical 
Skills 


Group Skills 


Variance due to Precollege Characteristics 
Unique Variance 3 due to: 


.056*** 


.024*** 


In-Class Experiences 


.072*** 


096*** 


Out-of-Class Experiences 


.031*** 


.012*** 


Total Shared Variance 


.017 b 


.013 b 


Total Variance Explained 


1 7 6*** 


. 14-5*** 


Adjusted R 2 


. 172*** 


. 142*** 



a Variance unattributable to any other variable set in the model. 
b Cannot be tested for statistical significance. 

***p < .001 









